Benchmarking Framework and Infrastructure Abstraction

Overview

This PR introduces an abstract benchmarking framework for vector databases and models, with reusable components and infrastructure abstractions for easy deployment on NRP. The framework code is provided by the imsearch_eval Python package, while this repository contains benchmark implementations and deployment infrastructure.

Framework Architecture

The benchmarking framework is provided by the imsearch_eval package.

Infrastructure Abstraction

Kubernetes Deployment

• Kustomize-based structure with base configurations and benchmark-specific overlays
• Vector DB and inference server agnostic: Base configurations work with any stack
• Each benchmark overlay adds environment variables for its specific services
• Template system for creating new benchmark deployments

Dockerfile System

• Template-based Dockerfiles for consistent container builds
• Separate containers for data loading and evaluation

Makefile System

• Base Makefile with common commands
• Benchmark-specific Makefiles that extend the base
• NRP-compatible commands and configurations

Template Directory

• Complete starter kit for new benchmarks
• Includes Makefile, Dockerfiles, Kubernetes configs, and Python templates
• Documentation and quick start guides

Repository Structure

benchmarking/
├── benchmarks/              # Benchmark implementations
│   ├── template/           # Template for creating new benchmarks
│   └── INQUIRE/           # INQUIRE benchmark implementation
│
└── kubernetes/             # Kubernetes deployment configurations
    ├── base/              # Base configurations (vector DB/inference server agnostic)
    └── INQUIRE/           # INQUIRE-specific overlay

Benefits

1. Faster benchmark creation using templates
2. Consistent evaluation across benchmarks via shared package
3. Easy integration of new vector databases and models
4. Maintainable shared code in separate package repository
5. NRP-ready deployment infrastructure
6. Framework stability independent of app/ changes
7. Flexible infrastructure supporting any vector DB/inference server combination
8. Easy distribution: Benchmarks can be used in any environment by installing the package

Future Work

This framework enables:

• Benchmarking Milvus, Pinecone, Qdrant, and other vector databases
• Adding new datasets with minimal code
• Comparing different models and vector databases
• Easy contribution of new adapters to the imsearch_eval package